Information processing device, map update method, program, and information processing system

ABSTRACT

There is provided an information processing device including: a global map acquiring unit that acquires at least a part of a global map representing positions of objects in a real space where a plurality of users are in activity; a local map generating unit that generates a local map representing positions of nearby objects detectable by a device of one user among the plurality of users; and an updating unit that updates the global map based on position data of objects included in the local map

CROSS REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/122,125 (filed Sep. 5, 2018), which is a continuation of U.S. patentapplication Ser. No. 15/647,813 (filed on Jul. 12, 2017 and issued asU.S. Pat. No. 10,083,188 on Sep. 25, 2018), which is a continuation ofU.S. patent application Ser. No. 15/253,188 (filed on Aug. 31, 2016 andissued as U.S. Pat. No. 9,727,580 on Aug. 8, 2017), which is acontinuation of U.S. patent application Ser. No. 14/691,197 (filed onApr. 20, 2015 and issued as U.S. Pat. No. 9,449,023 on Sep. 20, 2016),which is a continuation of U.S. patent application Ser. No. 13/037,788(filed on Mar. 1, 2011 and issued as U.S. Pat. No. 9,014,970 on Apr. 21,2015) which claims priority to Japanese Patent Application No.2010-051731 (filed on Mar. 9, 2010), which are all hereby incorporatedby reference in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an information processing device, a mapupdate method, a program, and an image processing system.

Description of the Related Art

Various applications for a plurality of users to share a maprepresenting positions of physical objects in the real space through anetwork are in practical use. As an example, there is an applicationthat allows a user to associate information such as a comment or aphotograph with a given position on a map and share the information orthe map with other users (cf. Japanese Patent Application Laid-Open No.2006-209784 and “Google Maps” (Internet URL: http://maps.google.com/)).Further, there is an application that associates a virtual tag with agiven position on a map and displays an image captured using a camerafunction of a terminal with the tag superimposed thereon (cf. “SekaiCamera Support Center” (Internet URL:http://support.sekaicamera.com/en)).

SUMMARY OF THE INVENTION

However, in the existing map share applications, although informationassociated with a map can be updated at will by a user, the map itselfdoes not change except for when a service provider updates it.Therefore, even when a user recognizes a change in the position of aphysical object in the real space, it is difficult to quickly reflectthe change on the map and share it with other users. Further, in aprivate space where a detailed map is not provided by a serviceprovider, it is difficult to share a map or information associated withthe map among users.

In light of the foregoing, it is desirable to provide novel and improvedinformation processing device, map update method, program, and imageprocessing system that enable a change in position of a physical objectin the real space to be quickly shared among users.

According to an embodiment of the present invention, there is providedan information processing device including: a global map acquiring unitthat acquires at least a part of a global map representing positions ofobjects in a real space where a plurality of users are in activity; alocal map generating unit that generates a local map representingpositions of nearby objects detectable by a device of one user among theplurality of users; and an updating unit that updates the global mapbased on position data of objects included in the local map.

The information processing device may further include: a calculatingunit that calculates a relative position of the local map to the globalmap based on position data of objects included in the global map andposition data of objects included in the local map; and a convertingunit that performs coordinate conversion from the position data ofobjects included in the local map to data of a coordinate system of theglobal map according to the relative position of the local map, whereinthe updating unit updates the global map by using the position data ofobjects included in the local map after the coordinate conversion by theconverting unit.

The information processing device may be a terminal device possessed bythe one user.

The global map acquiring unit may acquire at least a part of the globalmap from a server device storing the global map, and the updating unitmay update the global map of the server device by transmitting positiondata of objects to the server device.

The global map acquiring unit may acquire a part of the global mapcorresponding to a local area containing a position of the terminaldevice in the real space.

The global map acquiring unit may acquire a part of the global maprepresenting positions of a predetermined number of objects located inclose proximity to the terminal device.

The local map generating unit may generate the local map based on aninput image obtained by imaging the real space using an imaging deviceand feature data indicating a feature of appearance of one or moreobjects.

The calculating unit may calculate the relative position of the localmap based on position data of an immobile object included in common inthe global map and the local map.

The calculating unit may calculate the relative position of the localmap so that, when converting the position data of objects included inthe local map into data of the coordinate system of the global map, adifference between the data after conversion and the position data ofobjects included in the global map is smaller as a whole.

The global map may include position data of each object in the realspace in the coordinate system of the global map and a time stamprelated to the position data.

The information processing device may further include: a display controlunit that at least partially visualizes the global map onto a screen inresponse to an instruction from a user.

According to another embodiment of the present invention, there isprovided a map update method for updating a global map representingpositions of objects in a real space where a plurality of users are inactivity, performed by an information processing device, the methodincluding steps of: acquiring at least a part of the global map;generating a local map representing positions of nearby objectsdetectable by the information processing device; and updating the globalmap based on position data of objects included in the local map.

According to another embodiment of the present invention, there isprovided a program for causing a computer for controlling an informationprocessing device to function as: a global map acquiring unit thatacquires at least a part of a global map representing positions ofobjects in a real space where a plurality of users are in activity; alocal map generating unit that generates a local map representingpositions of nearby objects detectable by a device of one user among theplurality of users; and an updating unit that updates the global mapbased on position data of objects included in the local map.

According to another embodiment of the present invention, there isprovided an information processing system including: a server devicethat stores a global map representing positions of objects in a realspace where a plurality of users are in activity using a storage medium;and an information processing device possessed by one user among theplurality of users, the information processing device including a globalmap acquiring unit that acquires at least a part of the global map fromthe server device, a local map generating unit that generates a localmap representing positions of nearby objects detectable by theinformation processing device, and an updating unit that updates theglobal map based on position data of objects included in the local map.

According to the embodiments of the present invention described above,it is possible to provide the information processing device, the mapupdate method, the program, and the image processing system that enablea change in position of a physical object in the real space to bequickly shared among users.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram for illustrating an overview of a systemaccording to an embodiment;

FIG. 2 is a schematic diagram for illustrating position data of objectsincluded in a global map and a local map;

FIG. 3 is a block diagram illustrating an example of a configuration ofa server according to an embodiment;

FIG. 4 is an illustrative diagram for illustrating a partial global map;

FIG. 5 is a block diagram illustrating an example of a configuration ofa terminal device according to an embodiment;

FIG. 6A is an illustrative diagram for illustrating a first example of apartial global map acquisition process;

FIG. 6B is an illustrative diagram for illustrating a second example ofa partial global map acquisition process;

FIG. 6C is an illustrative diagram for illustrating a third example of apartial global map acquisition process;

FIG. 7 is a block diagram illustrating an example of a detailedconfiguration of a local map generating unit according to an embodiment;

FIG. 8 is a flowchart illustrating an example of a flow of aself-position detection process according to an embodiment;

FIG. 9 is an illustrative diagram for illustrating a feature point seton an object;

FIG. 10 is an illustrative diagram for illustrating addition of afeature point;

FIG. 11 is an illustrative diagram for illustrating an example of aprediction model;

FIG. 12 is an illustrative diagram for illustrating an example of aconfiguration of feature data;

FIG. 13 is a flowchart illustrating an example of a flow of an objectrecognition process according to an embodiment;

FIG. 14A is an illustrative diagram for illustrating an example of a mapmatching process by a calculating unit according to an embodiment;

FIG. 14B is an illustrative diagram for illustrating another example ofa map matching process by a calculating unit according to an embodiment;

FIG. 15A is an illustrative diagram for illustrating an example of aglobal map update process according to an embodiment;

FIG. 15B is an illustrative diagram for illustrating another example ofa global map update process according to an embodiment;

FIG. 16 is a flowchart illustrating an example of a flow of a map updateprocess between a map management server and a terminal device accordingto an embodiment;

FIG. 17 is a block diagram illustrating an example of a configuration ofa super client according to an embodiment; and

FIG. 18 is a schematic diagram illustrating an example of an applicationof sharing of additional information.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Preferred embodiments of the present invention will be describedhereinafter in the following order.

1. Overview of System

-   -   1-1. Example of System Configuration    -   1-2. Example of Position Data

2. Configuration of Map Management Server According to Embodiment

3. Configuration of Terminal Device According to Embodiment

-   -   3-1. Communication Interface    -   3-2. Imaging Unit    -   3-3 Initializing unit    -   3-4. Global Map Acquiring unit    -   3-5. Local Map Generating Unit    -   3-6. Calculating Unit    -   3-7. Converting Unit    -   3-8. Updating Unit    -   3-9. Display Control Unit

4. Flow of Process

5. Alternative Example

-   -   5-1. Super Client    -   5-2. Sharing of Additional Information

6. Summary

1. Overview of System [1-1. Example of System Configuration]

An overview of a system according to an embodiment of the presentinvention is described firstly with reference to FIGS. 1 and 2. FIG. 1is a schematic diagram illustrating an overview of an image processingsystem 1 according to an embodiment of the present invention. Referringto FIG. 1, the image processing system 1 according to the embodimentincludes a map management server 10, a terminal device 100 a and aterminal device 100 b.

The map management server 10 is an information processing device thatprovides a map share service for allowing a map and informationassociated with the map to be shared among a plurality of users. The mapmanagement server 10 has a database internally or externally and storesa global map, which is described later, in the database. The mapmanagement server 10 is typically implemented by using a general-purposeinformation processing device such as a personal computer (PC) or a workstation.

In this specification, a map managed by the map management server 10 isreferred to as a global map. The global map is a map that representspositions of physical objects in the real space all over a service areaA_(G) of the map share service.

The terminal device 100 a is an information processing device possessedby a user Ua. The terminal device 100 b is an information processingdevice possessed by a user Ub. In this specification, when there is noparticular need to distinguish between the terminal device 100 a and theterminal device 100 b, they are referred to collectively as the terminaldevice 100 by eliminating the alphabetical letter affixed to thereference numeral. The terminal device 100 can communicate with the mapmanagement server 10 through a communication connection by wire orwireless. The terminal device 100 may typically be an informationprocessing device of any type, such as PC, smart phone, personal digitalassistants (PDA), a portable music player or a game terminal.

The terminal device 100 has a sensor function capable of detectingpositions of nearby objects. Then, the terminal device 100 generates alocal map that represents positions of objects in the vicinity of itsown device (e.g. in an area A_(La) or an area A_(Lb)) using the sensorfunction. In this embodiment, a case of using Simultaneous localizationand mapping (SLAM) technology that can simultaneously estimate aposition and a posture of a camera and a position of a feature point ofan object present on an input image using a monocular camera as anexample of the sensor function is described.

Further, the terminal device 100 has an update function that updates theglobal map managed by the map management server 10 using the generatedlocal map and a display function that displays the latest global map (orthe global map at a certain point of time in the past). Specifically,the user Ua can view the global map updated by the terminal device 100 bpossessed by the user Ub on a screen of the terminal device 100 a, forexample. Further, the user Ub can view the global map updated by theterminal device 100 a possessed by the user Ua on a screen of theterminal device 100 b, for example.

[1-2. Example of Position Data]

FIG. 2 is a schematic diagram for illustrating position data of objectsincluded in a global map and a local map.

FIG. 2 shows four physical objects B1 to B4 present in the real space.The object B1 is a table. The object B2 is a coffee cup. The object B3is a notebook PC. The object B4 is a window. The position of the objectB4 usually does not move. In this specification, the object which doesnot move is referred to as an immobile object or a landmark. FIG. 2further shows position data R1 to R4 for the respective objects. Theposition data R1 to R4 include object ID “Obj1” to “Obj4”, position “X1”to “X4”, posture “Ω1” to “Ω4”, and time stamp “YYYYMMDDhhmmss”indicating the point of time when the position data is generated,respectively.

The global map is a data set that includes the position data asillustrated in FIG. 2 for physical objects present in the real space allover the service area A_(G). For example, when one entire building isthe service area A_(G), the global map can include position data of notonly the objects in one room illustrated in FIG. 2 but also objects inanother room. The coordinate system of the position data of the globalmap is previously set in a fixed manner as a global coordinate system.

On the other hand, the local map is a data set that includes theposition data as illustrated in FIG. 2 for physical objects present inthe real space in the vicinity of the terminal device 100. For example,the local map can include position data of the objects B1 to B4illustrated in FIG. 2. The position of the origin and the direction ofthe coordinate axis of the coordinate system of the local map depend onthe position and the posture of the camera of the terminal device 100.Therefore, the coordinate system of the local map is generally differentfrom the global coordinate system.

Note that objects whose positions can be represented by the global mapor the local map are not limited to the example shown in FIG. 2. Forexample, position data of objects such as a building or a car locatedoutdoors, instead of objects located indoors, may be included in theglobal map or the local map. In this case, the building may serve as alandmark.

2. Configuration of Map Management Server According to Embodiment

FIG. 3 is a block diagram illustrating an example of a configuration ofthe map management server 10 according to the embodiment. Referring goFIG. 3, the map management server 10 includes a communication interface20, a global map storage unit 30, a partial global map extracting unit40, an updating unit 50, and a global map delivery unit 60.

The communication interface 20 is an interface that mediates acommunication connection between the map management server 10 and theterminal device 100. The communication interface 20 may be a wirelesscommunication interface or a wired communication interface.

The global map storage unit 30 corresponds to a database that isconstructed using a storage medium such as a hard disk or asemiconductor memory and stores the above-described global maprepresenting positions of physical objects in the real space where aplurality of users are in activity. Then, the global map storage unit 30outputs a partial global map, which is a subset of the global map, inresponse to a request from the partial global map extracting unit 40.Further, the global map stored in the global map storage unit 30 isupdated by the updating unit 50. Further, the global map storage unit 30outputs the whole or a requested part of the global map in response to arequest from the global map delivery unit 60.

The partial global map extracting unit 40 receives information relatedto the position of the terminal device 100 through the communicationinterface 20 and extracts a partial global map according to theinformation. Then, the partial global map extracting unit 40 transmitsthe extracted partial global map to the terminal device 100 through thecommunication interface 20. The partial global map is a subset of theglobal map. The partial global map represents positions of physicalobjects located within a local area in the vicinity of the position ofthe terminal device 100 in the global coordinate system.

FIG. 4 is an illustrative diagram for illustrating the partial globalmap. A global map M_(G) that contains position data of 19 objects withthe object ID “Obj1” to “Obj19” is shown at the left of FIG. 4. The 19objects are scattered in the service area AC shown at the right of FIG.4. The objects whose distance from the position of the terminal device100 a possessed by the user Ua is a threshold D or less are objects B1to B9. In this case, the position data of the objects B1 to B9 form apartial global map M_(G)(Ua) for the user Ua, for example. Further, theobjects whose distance from the position of the terminal device 100 bpossessed by the user Ub is the threshold D or less are objects B11 toB19. In this case, the position data of the objects B11 to B19 form apartial global map M_(G)(Ub) for the user Lb, for example. The thresholdD is set to an appropriate in advance so that a large part of the rangeof the local map described later is included in the partial global map.Note that another example about extraction of the partial global map isfurther described later.

The updating unit 50 updates the global map stored in the global mapstorage unit 30 based on position data of objects received from theterminal device 100 through the communication interface 20. A change inposition of an object in the real space is thereby quickly reflected onthe global map. A global map update process by the updating unit 50 isfurther described later.

The global map delivery unit 60 delivers the global map stored in theglobal map storage unit 30 to the terminal device 100 in response to arequest from the the terminal device 100. The global map delivered fromthe global map delivery unit 60 is visualized on the screen of theterminal device 100 by a display function of the terminal device 100. Auser can thereby view the latest global map (or the global map at acertain point of time in the past).

3. Configuration of Terminal Device According to Embodiment

FIG. 5 is a block diagram illustrating an example of a configuration ofthe terminal device 100 according to the embodiment. Referring to FIG.5, the terminal device 100 includes a communication interface 102, animaging unit 110, an initializing unit 120, a global map acquiring unit130, a storage unit 132, a local map generating unit 140, a calculatingunit 160, a converting unit 170, an updating unit 180, and a displaycontrol unit 190.

[3-1. Communication Interface]

The communication interface 102 is an interface that mediates acommunication connection between the terminal device 100 and the mapmanagement server 10. The communication interface 102 may be a wirelesscommunication interface or a wired communication interface.

[3-2. Imaging Unit]

The imaging unit 110 may be realized as a camera having an imagingelement such as a charge coupled device (CCD) or a complementary metaloxide semiconductor (CMOS), for example. The imaging unit 110 may bemounted externally to the terminal device 100. The imaging unit 110outputs an image obtained by imaging the real space where physicalobjects are present as illustrated in FIG. 2 as an input image to theinitializing unit 120 and the local map generating unit 140.

[3-3. Initializing Unit]

The initializing unit 120 localizes a rough position of the terminaldevice 100 in the global coordinate system by using the input imageinput from the imaging unit 110. Localization of the position of theterminal device 100 based on the input image may be performed accordingto the technique disclosed in Japanese Patent Application Laid-Open No.2008-185417, for example. In this case, the initializing unit 120matches the input image against a reference image prestored in thestorage unit 132 and sets a high score to the reference image with ahigh degree of matching. Then, the initializing unit 120 calculates aprobability distribution of candidates for the position of the terminaldevice 100 based on the score and localizes the likely position (theposition with the highest probability value in the hypotheticalprobability distribution) of the terminal device 100 based on thecalculated probability distribution. Then, the initializing unit 120outputs the localized position of the terminal device 100 to the globalmap acquiring unit 130.

Note that the initializing unit 120 may localize the position of theterminal device 100 by using a global positioning system (GPS) functioninstead of the technique described above. Further, the initializing unit120 may localize the position of the terminal device 100 by using atechnique such as PlaceEngine capable of measuring the current positionbased on field intensity information from a wireless access point in thevicinity, for example.

[3-4. Global Map Acquiring Unit]

The global map acquiring unit 130 transmits information related to theposition of the terminal device 100 to the map management server 10through the communication interface 102 and acquires the above-describedpartial global map extracted by the partial global map extracting unit40 of the map management server 10. Then, the global map acquiring unit130 stores the acquired partial global map into the storage unit 132.

FIG. 6A is an illustrative diagram for illustrating a first example of apartial global map acquisition process. Referring to FIG. 6A, the globalmap acquiring unit 130 of the terminal device 100 transmits positioncoordinates of the terminal device 100 in the global coordinate systemto the map management server 10. Then, the partial global map extractingunit 40 of the map management server 10 extracts the partial global mapformed by position data of objects located within a radius D[m] from theposition coordinates, for example, and sends the extracted partialglobal map back to the terminal device 100. The global map acquiringunit 130 can thereby acquire the partial global map corresponding to thelocal area having a predefined width. The local area may be an areahaving a width which is directly observable by the imaging unit 110 ofthe terminal device 100, for example. This enables reduction ofcommunication costs and processing costs of the terminal device 100compared to the case of acquiring the whole global map.

FIG. 6B is an illustrative diagram for illustrating a second example ofthe partial global map acquisition process. Referring to FIG. 6B, theglobal map acquiring unit 130 of the terminal device 100 transmitsposition coordinates of the terminal device 100 in the global coordinatesystem to the map management server 10. Then, the partial global mapextracting unit 40 of the map management server 10 extracts the partialglobal map formed by position data of n-number of objects in ascendingorder of the distance from the position coordinates, for example, andsends the extracted partial global map back to the terminal device 100.The global map acquiring unit 130 can thereby acquire the partial globalmap corresponding to the local area containing a predefined number ofpieces of data. Generally, as the predefined number n of data isgreater, matching of the partial global map against the local map by thecalculating unit 160 to be described later can be made with higheraccuracy. The value of n is determined in consideration of the balancebetween accuracy of the matching and communication costs and processingcosts (for example, the value of n may be n=100).

FIG. 6C is an illustrative diagram for illustrating a third example ofthe partial global map acquisition process. Referring to FIG. 6C, theglobal map acquiring unit 130 of the terminal device 100 transmits anidentifier (hereinafter referred to as an area identifier) foridentifying an area where the terminal device 100 is located to the mapmanagement server 10. The area identifier may be an access pointidentifier of a wireless access point to which the terminal device 100can access, for example. Further, when one whole building is the servicearea A_(G), the area identifier may be a number identifying a floor orroom where the terminal device 100 is located. Receiving the areaidentifier, the partial global map extracting unit 40 of the mapmanagement server 10 extracts the partial global map formed by positiondata of objects contained in the area indicated by the area identifier,and sends the extracted partial global map back to the terminal device100. In this case also, compared to the case of acquiring the wholeglobal map, communication costs and processing costs of the terminaldevice 100 can be reduced.

[3-5. Local Map Generating Unit]

The local map generating unit 140 generates the above-described localmap that represents positions of nearby objects which are detectable bythe terminal device 100 based on an input image input from the imagingunit 110 and feature data, which is described later, stored in thestorage unit 132. FIG. 7 is a block diagram illustrating an example of adetailed configuration of the local map generating unit 140 according tothe embodiment. Referring to FIG. 7, the local map generating unit 140includes a self-position detecting unit 142, an image recognizing unit144 and a local map building unit 146.

(1) Self-Position Detecting Unit

The self-position detecting unit 142 dynamically detects a position of acamera, which takes input image, based on an input image input from theimaging unit 110 and feature data stored in the storage unit 132. Forexample, even when the camera of the imaging unit 110 is a monocularcamera, the self-position detecting unit 142 may dynamically determine aposition and a posture of the camera and a position of a feature pointon an imaging plane of the camera for each fame by applying the SLAMtechnology disclosed in “Real-Time Simultaneous Localization and Mappingwith a Single Camera” (Andrew J. Davison, Proceedings of the 9th IEEEInternational Conference on Computer Vision Volume 2, 2003, pp.1403-1410).

First, the entire flow of a self-position detection process by theself-position detecting unit 142 using the SLAM technology is describedwith reference to FIG. 8. Next, the self-position detection process isdescribed in detail with reference to FIGS. 9 to 11.

FIG. 8 is a flowchart illustrating an example of the flow of theself-position detection process by the self-position detecting unit 142using the SLAM technology. In FIG. 8, when the self-position detectionprocess starts, the self-position detecting unit 142 first initializes astate variable (step S102). In this embodiment, the state variable is avector including a position and a posture (rotation angle) of thecamera, a moving speed and an angular speed of the camera, and aposition of one or more feature points as elements. The self-positiondetecting unit 142 then sequentially obtains the input image from theimaging unit 110 (step S112). The processes from the step 112 to thestep S118 may be repeated for each input image (that is, each frame).

At the step S114, the self-position detecting unit 142 tracks featurepoints present in the input image. For example, the self-positiondetecting unit 142 detects a patch (small image of 3×3=9 pixels around afeature point, for example) of each feature point stored in advance inthe storage unit 132 from the input image. The position of the patchherein detected, that is, the position of the feature point is usedlater when updating the state variable.

At the step S116, the self-position detecting unit 142 generates apredicted value of the state variable of next frame, for example, basedon a predetermined prediction model. Also, at the step S18, theself-position detecting unit 142 updates the state variable using thepredicted value of the state variable generated at the step S116 and anobserved value according to the position of the feature point detectedat the step S114. The self-position detecting unit 142 executes theprocesses at the steps S116 and S118 based on a principle of an extendedKalman filter.

As a result of such process, a value of the state variable updated foreach frame is output. Configuration of each process of tracking of thefeature point (step S114), prediction of the state variable (step S116)and updating of the state variable (step S118) are hereinafter describedmore specifically.

(1-1) Tracking of Feature Point

In this embodiment, the storage unit 132 stores in advance the featuredata indicating features of objects corresponding to physical objectswhich may be present in the real space. The feature data includes smallimages, that is, the patches regarding one or more feature points, eachrepresenting the feature of appearance of each object, for example. Thepatch may be the small image composed of 3×3=9 pixels around the featurepoint, for example.

FIG. 9 illustrates two examples of the objects and an example of featurepoints (FPs) and patches set on each object. A left object in FIG. 9 isthe object representing a PC (refer to FIG. 5a ). A plurality of featurepoints including a feature point FP1 are set on the object. Further, apatch Pth1 is defined in relation to the feature point FP1. On the otherhand, a right object in FIG. 9 is the object representing a calendar(refer to FIG. 9b ). A plurality of feature points including a featurepoint FP2 are set on the object. Further, a patch Pth2 is defined inrelation to the feature point FP2.

Upon obtaining an input image from the imaging unit 110, theself-position detecting unit 142 matches partial images included in theinput image against the patch for each feature point illustrated in FIG.9 stored in advance in the storage unit 132. The self-position detectingunit 142 then specifies a position of each feature point included in theinput image (a position of a center pixel of the detected patch, forexample) as a result of the matching.

It should be noted that, for tracking feature points (step S114 in FIG.8), it is not necessary to store data regarding all of the featurepoints to be tracked in the storage unit 132 in advance. For example,four feature points are detected in the input image at time T=t−1 in anexample illustrated in FIG. 10 (refer to FIG. 10a ). Next, when theposition or the posture of the camera changes at time T=t, only two ofthe four feature points present in the input image at the time T=t−1 arepresent in the input image. In this case, the self-position detectingunit 142 may newly set feature points at positions where acharacteristic pixel pattern of the input image is present and use thenew feature points in the self-position detection process for asubsequent frame. For example, in the example illustrated in FIG. 10,three new feature points are set on the object at the time T=t (refer toFIG. 10b ). This is a feature of the SLAM technology, and according tothis, a cost of setting all of the feature points in advance may bereduced and accuracy of the process may be improved using the increasednumber of feature points.

(1-2) Prediction of State Variable

In this embodiment, the self-position detecting unit 142 uses a statevariable X expressed in the following equation as the state variable tobe applied for the extended Kalman filter.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack & \; \\{X = \begin{pmatrix}x \\\omega \\\overset{.}{x} \\\overset{.}{\omega} \\p_{1} \\\vdots \\p_{N}\end{pmatrix}} & (1)\end{matrix}$

The first element of the state variable X in the equation (1) representsa three-dimensional position of the camera in a local map coordinatesystem (x, y, z) which is set for each terminal device 100. Note thatthe local map coordinate system is set according to the position and theposture of the camera of the imaging unit 110 in the initializationprocess, for example. The origin of the local map coordinate system maybe at the position of the camera in the initialization process, forexample.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack & \; \\{x = \begin{pmatrix}x_{c} \\y_{c} \\z_{c}\end{pmatrix}} & (2)\end{matrix}$

Also, the second element of the state variable is a four-dimensionalvector to having a quaternion as an element corresponding to a rotationmatrix representing the posture of the camera. Note that, the posture ofthe camera may be represented using an Euler angle in place of thequaternion. Also, the third and the fourth elements of the statevariable represent the moving speed and the angular speed of the camera,respectively.

Further, the fifth and subsequent elements of the state variablerepresent a three-dimensional position p_(i) of a feature point FP_(i)(i=1 . . . N) in the local map coordinate system as expressed in afollowing equation. Note that, as described above, the number N of thefeature points may change during the process.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack & \; \\{p_{i} = \begin{pmatrix}x_{i} \\y_{i} \\z_{i}\end{pmatrix}} & (3)\end{matrix}$

The self-position detecting unit 142 generates the predicted value ofthe state variable for the latest frame based on the value of the statevariable X initialized at the step S102 or the value of the statevariable X updated in the previous frame. The predicted value of thestate variable is generated according to a state equation of theextended Kalman filter according to multidimensional normal distributionas shown in the following equation.

[Equation 4]

Predicted state variable {circumflex over (X)}=F(X,a)+w  (4)

Herein, F represents the prediction model regarding state transition ofa system and “a” represents a prediction condition. Also, w representsGaussian noise and may include a model approximation error, anobservation error and the like, for example. In general, an average ofthe Gaussian noise w is 0.

FIG. 11 is an illustrative diagram for illustrating an example of theprediction model according to this embodiment. With reference to FIG.11, two prediction conditions in the prediction model according to thisembodiment are illustrated. First, as a first condition, suppose thatthe three-dimensional position of the feature point in the local mapcoordinate system does not change. That is, provided that thethree-dimensional position of the feature point FP1 at the time T isp_(T), the following relationship is satisfied.

[Equation 5]

p _(t) =p _(t−1)  (5)

Next, as a second condition, suppose that motion of the camera isuniform motion. That is, a following relationship is satisfied for thespeed and the angular speed of the camera from the time T=t−1 to thetime T=t.

[Equation 6]

{dot over (x)} _(i) ={dot over (x)} _(t−1)  (6)

{dot over (ω)}_(t)={dot over (ω)}_(t−1)  (7)

The self-position detecting unit 142 generates the predicted value ofthe state variable for the latest frame based on such prediction modeland the state equation expressed in the equation (4).

(1-3) Updating of State Variable

The self-position detecting unit 142 then evaluates an error betweenobservation information predicted from the predicted value of the statevariable and actual observation information obtained as a result offeature point tracking, using an observation equation, for example. Notethat, v in the equation (8) is the error.

[Equation 7]

Observation information s=H({circumflex over (x)})+v  (8)

Predicted observation information ŝ=H({circumflex over (X)})  (9)

Herein, H represents an observation model. For example, a position ofthe feature point FP_(i) on the imaging plane (u-v plane) is defined asexpressed in a following equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack & \; \\{{{Position}\mspace{14mu} {of}\mspace{14mu} {FP}_{i}\mspace{14mu} {on}\mspace{14mu} {imaging}\mspace{14mu} {plane}\mspace{14mu} \overset{\sim}{p_{i}}} = \begin{pmatrix}u_{i} \\v_{i} \\1\end{pmatrix}} & (10)\end{matrix}$

Herein, all of the position of the camera x, the posture of the camera ωand the three-dimensional position p_(i) of the feature point FP_(i) aregiven as the elements of the state variable X. Then, the position of thefeature point FP_(i) on the imaging plane is derived using a followingequation according to a pinhole model.

[Equation 9]

λ{tilde over (p)} _(i) =AR _(ω)(p _(i) −x)  (11)

Herein, λ represents a parameter for normalization, A represents acamera internal parameter, R_(ω) represents the rotation matrixcorresponding to the quaternion ω representing the posture of the cameraincluded in the state variable X. The camera internal parameter A isgiven in advance as expressed in the following equation according tocharacteristics of the camera, which takes the input image.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 10} \right\rbrack & \; \\{A = \begin{pmatrix}{{- f} \cdot k_{u}} & {{f \cdot k_{u} \cdot \cot}\; \theta} & u_{O} \\0 & {- \frac{f \cdot k_{v}}{\sin \; \theta}} & v_{O} \\0 & 0 & 1\end{pmatrix}} & (12)\end{matrix}$

Herein, f represents focal distance, θ represents orthogonality of animage axis (ideal value is 90 degrees), k_(u) represents a scale along alongitudinal axis of the imaging plane (rate of change of scale from thelocal map coordinate system to the coordinate system of the imagingplane), k_(v) represents a scale along an abscissa axis of the imagingplane, and (u₀, v₀) represents a center position of the imaging plane.

Therefore, a feasible latest state variable X may be obtained bysearching the state variable X, which makes the error between thepredicted observation information derived using the equation (11), thatis, the position of each feature point on the imaging plane and theresult of feature point tracking at the step S114 in FIG. 8 minimum.

[Equation 11]

Latest state variable X←{circumflex over (X)}+Innov(s−ŝ)  (13)

The self-position detecting unit 142 outputs the position x and theposture M of the camera dynamically updated by applying the SLAMtechnology in this manner to the local map building unit 146.

(2) Feature Data

The storage unit 132 stores in advance the feature data indicatingfeatures of objects corresponding to physical objects which may bepresent in the real space. FIG. 12 is an illustrative diagram forillustrating an example of a configuration of feature data.

Referring to FIG. 12, feature data FD1 as an example of the object B2 isillustrated. The feature data FD1 includes an object name FD11, imagedata FD12 taken from six directions, patch data FD13, three-dimensionalshape data FD14, and ontology data FD15.

The object name FD11 is a name that can specify a corresponding object,such as “coffee cup A”.

The image data FD12 includes six image data obtained by taking images ofthe corresponding object from six directions (front, back, left, right,above and below), for example. The patch data FD13 is a set of smallimages around each feature point for each of one or more feature pointsset on each object. The image data FD12 and the patch data FD13 may beused for an object recognition process by the image recognizing unit 144to be described later. Also, the patch data FD13 may be used for theabove-described self-position detection process by the self-positiondetecting unit 142.

The three-dimensional shape data FD14 includes polygon information forrecognizing a shape of the corresponding object and three-dimensionalpositional information of feature points. The three-dimensional shapedata FD14 may be used for a local map build process by the local mapbuilding unit 146 to be described later.

The ontology data FD15 is the data, which may be used to assist thelocal map build process by the local map building unit 146, for example.In the example illustrated in FIG. 12, the ontology data FD15 indicatesthat the object B2, which is the coffee cup, is likely to come incontact with an object corresponding to a table and is unlikely to comein contact with an object corresponding to a bookshelf.

(3) image Recognizing Unit

The image recognizing unit 144 specifies to which object each of theobjects present on the input image corresponds by using theabove-described feature data stored in the storage unit 132.

FIG. 13 is a flowchart illustrating an example of flow of the objectrecognition process by the image recognizing unit 144. Referring to FIG.13, the image recognizing unit 144 first obtains the input image fromthe imaging unit 110 (step S212). Next, the image recognizing unit 144matches partial images included in the input image against patches ofone or more feature points of each object included in the feature datato extract feature points included in the input image (step S214). Itshould be noted that the feature points used in the object recognitionprocess by the image recognizing unit 144 and the feature points used inthe self-position detection process by the self-position detecting unit142 are not necessarily the same. However, when common feature pointsare used in the both processes, the image recognizing unit 144 may reusethe result of feature point tracking by the self-position detecting unit142.

Next, the image recognizing unit 144 specifies the object present in theinput image based on an extraction result of the feature point (stepS216). For example, when the feature points belonging to one object areextracted with high density in a certain area, the image recognizingunit 144 may recognize that the object is present in the area. The imagerecognizing unit 144 then outputs the object name (or identifier) of thespecified object and the position of the feature point belonging to theobject on the imaging plane to the local map building unit 146 (stepS218).

(4) Local Map Building Unit

The local map building unit 146 generates the local map using theposition and the posture of the camera input from the self-positiondetecting unit 142, the positions of the feature points on the imagingplane input from the image recognizing unit 144 and the feature datastored in the storage unit 132. In this embodiment, the local map is aset of position data indicating positions and postures of one or moreobjects present in the vicinity of the terminal device 100 using thelocal map coordinate system. Further, the respective position dataincluded in the local map may be associated with object namescorresponding to objects, the three-dimensional positions of featurepoints belonging to the objects, and the polygon information configuringshapes of the objects, for example. The local map may be built byobtaining the three-dimensional position of each feature point accordingto the above-described pinhole model from the position of the featurepoint on the imaging plane input from the image recognizing unit 144,for example.

By deforming the relation equation of the pinhole model expressed in theequation (11), the three-dimensional position p_(i) of the feature pointFP_(i) in the local map coordinate system may be obtained by a followingequation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 12} \right\rbrack & \; \\{p_{i} = {{x + {\lambda \cdot R_{\omega}^{T} \cdot A^{- 1} \cdot {\overset{\sim}{p}}_{i}}} = {x + {{d \cdot R_{\omega}^{T}}\frac{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}{{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}}}}}} & (14)\end{matrix}$

Herein, d represents distance between the camera and each feature pointin the local map coordinate system. The local map building unit 146 maycalculate such distance d based on the positions of at least fourfeature points on the imaging plane and the distance between the featurepoints for each object. The distance between the feature points isstored in advance in the storage unit 132 as the three-dimensional shapedata FD14 included in the feature data illustrated with reference toFIG. 12. It should be noted that, a calculation process of the distanced in the equation (14) is disclosed in detail in Japanese PatentApplication Laid-Open No. 2008-304268.

After the distance d is calculated, remaining variables of a right sideof the equation (14) are the position and the posture of the camerainput from the self-position detecting unit 142 and the position of thefeature point on the imaging plane input from the image recognizing unit144, and all of which are known. The local map building unit 146 thencalculates the three-dimensional position in the local map coordinatesystem for each feature point input from the image recognizing unit 144according to the equation (14). The local map building unit 146 thenbuilds a latest local map according to the three-dimensional position ofeach calculated feature point. It should be noted that, at that time,the local map building unit 146 may improve accuracy of the data of thelocal map using the ontology data F5 included in the feature dataillustrated with reference to FIG. 12. Then, the local map building unit146 stores the built local map into the storage unit 132 and outputs thelocal map to the calculating unit 160.

It should be noted that the case where the local map generating unit 140generates the local map by using the image (such as the patch for eachfeature point of the object) included in the feature data is describedabove. However, when an identifying means such as a marker or a tag foridentifying each object is provided in advance to the object, the localmap generating unit 140 may identify each object using the identifyingmeans and generate the local map using the identification result. As theidentifying means for identifying each object, a barcode, QR code(registered trademark), an RF (Radio Frequency) tag and the like may beused, for example.

[3-6. Calculating Unit]

The calculating unit 160 matches the position data of objects includedin the partial global map against the position data of objects includedin the local map and, based on a result of the matching, calculates therelative position and posture of the local map relative to the globalmap. The relative position and posture of the local map to the partialglobal map correspond to the displacement and slope of the local mapcoordinate system with reference to the global coordinate system.Specifically, the calculating unit 160 may calculate the relativeposition and posture of the local map based on the position data of thelandmark included in common in the partial global map and the local map,for example. Alternatively, the calculating unit 160 may calculate therelative position and posture of the local map so that, when theposition data of objects included in the local map is converted intodata of the global coordinate system, a difference between the converteddata and the position data of objects included in the partial global mapbecomes small as a whole, for example. Then, the calculating unit 160outputs the calculated relative position and posture of the local mapand the local map to the converting unit 170.

(First Technique)

FIG. 14A is an illustrative diagram for illustrating an example of a mapmatching process by the calculating unit 160 according to theembodiment.

FIG. 14A schematically shows a partial global map M_(G) (Ua) stored inthe storage unit 132 and a local map M_(La) generated by the local mapgenerating unit 140. The partial global map M_(G) (Ua) includes nineobjects represented by cubes. Further, the local map M_(La) includesfour objects represented by spheres. Of those objects, objects B1,B2(B2′), B3 and B4 are included in common in the partial global mapM_(G) (Ua) and the local map M_(La). The objects B1, B3 and B4 areimmobile objects which hardly move, that are landmarks. The calculatingunit 160 thus calculates a position and a posture of the local map insuch a way that, when the position data of the objects B1, B3 and B4 ofthe local map is converted into data of the global coordinate system,the converted data and the position data of the objects B1, B3 and B4included in the partial global map coincide with each other, forexample.

First, the calculating unit 160 extracts three pairs of landmarks (orfeature points on the landmarks) common to the partial global map M_(G)(Ua) and the local map M_(La). For the three pairs of landmarks, athree-dimensional position in the local map coordinate system is w_(i)(i=1, 2, 3), and a three-dimensional position in the global coordinatesystem is X_(i). Further, when a displacement of the local mapcoordinate system with reference to the global coordinate system is ΔX,and a rotation matrix corresponding to a slope ΔΩ of the local mapcoordinate system is R_(L), the following equations are established.

[Equation 13]

X ₁ =R _(L) ·w ₁ +ΔX

X ₂ =R _(L) ·w ₂ +ΔX

X ₃ =R _(L) ·w ₃ +ΔX  (5)

The rotation matrix R_(L) may be obtained by altering the equation (15)into the following equation (16).

[Equation 14]

X ₁ −X ₃ =R _(L)·(w ₁ −w ₃)

X ₂ −X ₃ =R _(L)·(w ₂ −w ₃)

(X ₁ −X ₃)×(X ₂ −X ₃)=R _(L){(w ₁ −w ₃)×(w ₂ −w ₃)}  (16)

It should be noted that, in the case of representing the rotation of thelocal map coordinate system using a quaternion, instead the rotationmatrix R_(L), as well, a quaternion representing a rotation (and arotation matrix corresponding to the quaternion) may be calculated usingthat the norm of the quaternion indicating the rotation is 1.

Further, the calculating unit 160 can calculate the displacement ΔX ofthe local map coordinate system by solving the equations (15).

Note that, in the example of FIG. 14A, the position of the object B2,which is not a landmark, in the partial global map M_(G) (Ua) and theposition (B2′) in the local map M_(La) are different. This means thatthe object B2 has moved during a period from the latest update time ofthe partial global map M_(G) (Ua) to the generation time of the localmap M_(La).

(Second Technique)

FIG. 14B is an illustrative diagram for illustrating another example ofthe map matching process by the calculating unit 160 according to theembodiment.

FIG. 14 schematically shows a partial global map M_(G) (Ua) stored inthe storage unit 132 and a local map M_(La) generated by the local mapgenerating unit 140. The partial global map M_(G) (Ua) includes nineobjects represented by cubes. Further, the local map M_(La) includesfour objects represented by spheres. Of those objects, objects B5, B6,B7 and B8 are included in common in the partial global map M_(G) (Ua)and the local map M_(La). The calculating unit 160 may thus calculatethe position (the displacement ΔX of the local map coordinate system)and the posture (the slope ΔΩ of the local map coordinate system) of thelocal map in such a way that, when the position data of the objects B5to B8 is converted into data of the global coordinate system, the totalsum of differences between the converted data and the position data ofthe objects B5 to B8 (e.g. the sum of distances E5, E6, E7 and E8 in theexample of FIG. 14A) is small, for example. For example, the calculatingunit 160 may calculate the likely position and posture of the local mapat which the above-described total sum of differences is small byapplying known RANdom Sample Consensus (RANSAC) algorithm.

The RANSAC algorithm is generally an algorithm that, for a plurality ofpoints whose three-dimensional positions are known, decides a line or aplane containing the plurality of points within a preset range of error.In this embodiment, the calculating unit 160 first prepares a pluralityof (suitably, four or more) pairs of objects (or feature points onobjects) which are common to the partial global map M_(G) (Ua) and thelocal map M_(La). Next, the calculating unit 160 extracts three pairsfrom the prepared plurality of pairs at random. Then, the calculatingunit 160 derives candidates for the displacement ΔX of the local mapcoordinate system and the rotation matrix R_(L) corresponding to theslope ΔΩ of the local map coordinate system by solving theabove-described equations (15). Then, the calculating unit 160 evaluatesthe derived candidates using the following evaluation formula.

[Equation 15]

ε_(j) =∥X _(j)−(R _(L) ·w _(j) +ΔX)∥  (17)

Note that, in the equation (17), j indicates each of objects (or featurepoints on objects). The calculating unit 160 counts the number of jwhich makes an evaluation value ε_(j) in the equation (17) smaller thana predetermined threshold. Then, the calculating unit 160 decides thedisplacement ΔX and the rotation matrix R_(L) by which the count resultis the greatest as the likely position and posture of the local map.

(Combination of First Technique and Second Technique)

Further, the calculating unit 160 may use both the first technique andthe second technique described above as appropriate according to thenumber of landmarks included in the local map, for example. For example,the relative position and posture of the local map may be calculatedusing the technique described with reference to FIG. 14A when threelandmarks are included in the local map and calculated using thetechnique described with reference to FIG. 14B when two or lesslandmarks are included in the local map.

[3-7. Converting Unit]

The converting unit 170 performs coordinate conversion of position dataof objects included in the local map into data of the global mapcoordinate system according to the relative position and posture of thelocal map input from the calculating unit 160. Specifically, theconverting unit 170 rotates the three-dimensional position of objectsincluded in the local map (local map coordinate system) using therotation matrix corresponding to the slope ΔΩ of the local map inputfrom the calculating unit 160. Then, the converting unit 170 adds therelative position (the displacement ΔX of the local map coordinatesystem with respect to the global coordinate system) of the local mapinput from the calculating unit 160 to the coordinates after rotation.The position data of objects included in the local map is therebyconverted into data of the global map coordinate system. The convertingunit 170 outputs the position data of objects included in the local mapafter the coordinate conversion to the updating unit 180.

Further, the converting unit 170 may perform coordinate conversion ofthe position and posture of the camera of the local map coordinatesystem detected by the self-position detecting unit 142 of the local mapgenerating unit 140 into data of the global map coordinate system usingthe relative position and posture of the local map input from thecalculating unit 160. The position of the terminal device 100 which isspecified by the initializing unit 120 can be thereby updated accordingto movement of the terminal device 100 after initialization. After that,the global map acquiring unit 130 may acquire a new partial global mapfrom the map management server 10 according to the new updated positionof the terminal device 100.

[3-8. Updating Unit]

The updating unit 180 updates the partial global map stored in thestorage unit 132 by using the position data of objects included in thelocal map after the coordinate conversion by the converting unit 170.Further, the updating unit 180 transmits the local map after thecoordinate conversion by the converting unit 170 or the updated partialglobal map to the map management server 10, thereby updating the globalmap stored in the map management server 10. The update of the global mapcan be finally performed by the updating unit 50 of the map managementserver 10 which has received the local map after the coordinateconversion or the global map after the update from the updating unit 180of the terminal device 100.

FIG. 1A is an illustrative diagram for illustrating an example of aglobal map update process according to the embodiment.

FIG. 15A schematically shows a partial global map M_(G) (Ua) beforeupdate and a local map M_(La)′ after coordinate conversion. The partialglobal map M_(G) (Ua) before update includes position data of nineobjects (Obj1 to Obj9). The time stamp of those position data indicates2010/1/1 23:59:59 as an example. The local map M_(La)′ after coordinateconversion includes position data of five objects (Obj1 to Obj4). Thetime stamp of those position data indicates 2010/1/2 12:00:00 as anexample. In this case, the updating unit 180 updates the position dataof the partial global map M_(G) (Ua) to the position data of the morerecent local map M_(La)′ after coordinate conversion for each of thecommon objects (Obj1 to Obj4). Further, the updating unit 180 insertsposition data of a new object (Obj10) into the partial global map M_(G)(Ua). As a result, a new partial global map M_(G) (Ua)′ is generated.

Note that, although the position data of common objects are updated bynew data in the example of FIG. 15A, old position data and new positiondata of the common objects may coexist instead. This enables extractionby filtering not only the latest global map but also the past global mapwith time. In this case, the old position data may be deleted after acertain period of time has elapsed by periodic processing, for example.

FIG. 15B is an illustrative diagram for illustrating another example ofa global map update process according to the embodiment.

FIG. 15B schematically shows a partial global map M_(G)(Ua) beforeupdate and a local map M_(La)′ after coordinate conversion. The partialglobal map M_(G) (Ua) before update includes seven objects includingobjects B to B3. On the other hand, the local map M_(La)′ aftercoordinate conversion includes four objects B1 to B3 and B12. In thiscase, the updating unit 180 deletes objects in the partial global mapM_(G) (Ua) before update which are included in an area corresponding tothe local map M_(La)′ after coordinate conversion (the area indicated bya dotted line in FIG. 15B), for example, and inserts the objects in thelocal map M_(La)′ after coordinate conversion which are included in thisarea into the partial global map M_(G) (Ua). As a result, a new partialglobal map M_(G) (Ua)′ as shown at the bottom of FIG. 15B is generated.According to such update process, it is possible to appropriatelyreflect movement of a plurality of objects which are difficult to bedistinguished by image recognition, such as a plurality of objectshaving the same appearance, on the global map.

Note that, in the example of FIG. 15B also, old position data and newposition data may coexist in one map. Further, because objects servingas landmarks among the objects included in the partial global map do notusually move, the updating unit 180 may skip the update process for theposition data of the landmarks.

The above description with reference to FIGS. 15A and 15B is alsoapplicable in the same manner to update of the global map by theupdating unit 50 of the map management server 10, not only update of thepartial global map by the updating unit 180 of the terminal device 100.In this case, the partial global map in FIGS. 15A and 15B corresponds tothe global map stored in the global map storage unit 30, and the localmap after coordinate conversion in FIGS. 15A and 15B corresponds to thepartial global map (or the local map after coordinate conversion)transmitted from the terminal device 100 to the map management server10.

[3-9. Display Control Unit]

The display control unit 190 downloads the global map from the mapmanagement server 10 in response to an instruction from a user,visualizes the global map at least partially, and outputs it to a screenof the terminal device 100. Specifically, when the display control unit190 detects input of an instruction from a user, for example, thedisplay control unit 190 transmits a request for transmission of theglobal map to the global map delivery unit 60 of the map managementserver 10. Then, the global map stored in the global map storage unit 30is delivered from the global map delivery unit 60 of the map managementserver 10. The display control unit 190 receives the global map,visualizes the positions of objects in an area desired by a user (whichmay be an area different from the area where a user is currentlylocated), and outputs it to the screen.

4. Flow of Process

FIG. 16 is a flowchart illustrating an example of a flow of a map updateprocess between the map management server 10 and the terminal device 100according to the embodiment.

Referring to FIG. 16, the initializing unit 120 of the terminal device100 first initializes the position of the terminal device 100 in theglobal coordinate system by using an input image input from the imagingunit 110 (step S302). The initialization process by the initializingunit 120 may be performed at the startup of the terminal device 100, atthe startup of a given application in the terminal device 100 and thelike, for example.

Next, the global map acquiring unit 130 of the terminal device 100transmits information related to the position of the terminal device 100in the global coordinate system to the map management server 10 (stepS304). The information related to the position of the terminal device100 may be coordinates of the terminal device 100 in the globalcoordinate system or may alternatively be an area identifier foridentifying the area where the terminal device 100 is located.

Then, the partial global map extracting unit 40 of the map managementserver 10 extracts the partial global map for the terminal device 100,which is a subset of the global map stored in the global map storageunit 30 (step S306).

The partial global map extracting unit 40 of the map management server10 then transmits the partial global map for the terminal device 100 tothe global map acquiring unit 130 of the terminal device 100 (stepS308).

Then, the local map generating unit 140 of the terminal device 100generates the local map representing positions of nearby objects basedon an input image and feature data (step S310).

After that, the calculating unit 160 of the terminal device 100calculates the relative position and posture of the local map on thebasis of the global coordinate system based on the position data ofobjects included in the partial global map and the position data ofobjects included in the local map (step S312). Then, according to therelative position and posture of the local map calculated by thecalculating unit 160, the converting unit 170 performs coordinateconversion of the position data of objects included in the local mapinto data of the global coordinate system.

Then, the updating unit 180 of the terminal device 100 updates thepartial global map stored in the storage unit 132 of the terminal device100 by using the position data of objects included in the local mapafter coordinate conversion. Further, the updating unit 180 updates theposition of the terminal device 100 in the global coordinate system(step S314).

Then, the updating unit 180 of the terminal device 100 transmits theupdated partial global map to the updating unit 50 of the map managementserver 10 (step S316). The updating unit 50 of the map management server10 then updates the global map stored in the global map storage unit 30by using the position data of objects included in the updated partialglobal map (step S318).

After that, the process from the generation of the local map (step S310)to the update of the global map (step S318) is repeated on a regularbasis or in response to a request. Further, when the position of theterminal device 100 becomes apart from the center of the partial globalmap by a predetermined distance or more, for example, the process fromthe acquisition of the partial global map (step S304) based on thelatest position of the terminal device 100 can be performed.

A user can view the global map which is updated by such map updateprocess on the screen of the terminal device 100 by using the displayfunction provided through the global map delivery unit 60 of the mapmanagement server 10 and the display control unit 190 of the terminaldevice 100.

5. Alternative Example [5-1. Super Client]

The case where the map update process is performed between the mapmanagement server 10 and the terminal device 100 is mainly describedabove. However, the present invention is not limited thereto. Forexample, the terminal device 100 possessed by any user may have a globalmap management function similar to that of the map management server 10.Such terminal device is referred to as a super client in this section.

FIG. 17 is a block diagram illustrating an example of a configuration ofa super client 200 as an example. Referring to FIG. 17, the super client200 includes a communication interface 210, a server processing module220, a client processing module 230 and a display control unit 240.

The communication interface 210 is an interface that mediates acommunication connection between the super client 200 and anotherterminal device (for example, the terminal device 100 illustrated inFIG. 5). The communication interface 210 may be a wireless communicationinterface or a wired communication interface.

The server processing module 220 is a module that has a global mapmanagement function similar to that of the map management server 10described above. The server processing module 220 includes the globalmap storage unit 30, the partial global map extracting unit 40, theupdating unit 50 and the global map delivery unit 60. The serverprocessing module 220 performs the map update process as described withreference to FIG. 16 with another terminal device and further performsthe map update process also with the client processing module 230 to bedescribed later.

The client processing module 230 is a module that has a global mapupdate and viewing function and the like similar to that of the mapmanagement server 10 described above. The client processing module 230includes the imaging unit 110, the initializing unit 120, the global mapacquiring unit 130, the storage unit 132, the local map generating unit140, the calculating unit 160, the converting unit 170 and the updatingunit 180.

The display control unit 240, in response to an instruction from a user,at least partially visualizes the global map stored in the global mapstorage unit 30 of the server processing module 220 and outputs it to ascreen. Further, the display control unit 240 may visualize the partialglobal map or the local map stored in the storage unit 132 of the clientprocessing module 230 and output it to the screen.

Use of the above-described super client 200 and one or more terminaldevice 100 enables a plurality of users to share one global map througha communication connection using P2P (Peer to Peer), for example.

[5-2. Sharing of Additional Information]

The terminal device 100 may transmit additional information such as animage captured using the imaging unit 110 and the like, for example, inassociation with coordinate data together with the partial global map tothe map management server 10. Then, the additional information is madeviewable through the global map delivery unit 60 of the map managementserver 10 and the display control unit 190 of the terminal device 100,thereby enabling a plurality of users to share the additionalinformation associated with the global map which is updated dynamically.The additional information may be the actual image captured in theterminal device 100 or a recognition result such as characters presenton the image, for example. For example, by sharing reading results ofthe numbers of cars parked in the streets in association with positiondata of the cars in the global map, it can be used for detection oftraffic violation in no-parking zones and the like (cf. FIG. 18).Further, by sharing price information of products in each store ofshopping streets in association with position data of the products inthe global map, it is expected to improve the convenience for shoppingby users and to promote the sales of the products. Furthermore, bysharing data of stocks in a warehouse in association with position dataof the products in the global map, stock control is facilitated.

6. Summary

One embodiment of the present invention and its application aredescribed above with reference to FIGS. 1 to 18. According to theembodiment, in the information processing device such as the terminaldevice 100 or the super client 200, the partial global map, which is atleast a part of the global map representing positions of physicalobjects present in the real space where a plurality of users are inactivity, is acquired, and the local map representing positions ofnearby objects detectable by the device is generated. Then, the partialglobal map and the global map shared among a plurality of users areupdated based on the position data of objects included in the local map.It is thereby possible to quickly share a change in position of anobject in the real space among users. Further, because the position dataof objects in the vicinity of the terminal device 100 which are detectedin the local map generation process can be added to the global map, evenin the space where a detailed map is not provided by a service provider,more detailed position data of objects can be shared among users.

According to the embodiment, after the relative position and posture ofthe local map on the basis of the global coordinate system arecalculated based on the position data of objects included in the partialglobal map and the position data of objects included in the local map,the above-described update process is performed using the position dataof the local map which is coordinate-converted according to thecalculation result. Therefore, no restrictions are placed on the localmap coordinate system, and therefore a user can freely carry theinformation processing device and share the dynamically updated globalmap with other users.

Further, according to the embodiment, the above-described partial globalmap is acquired from the server device that stores the global map. Thepartial global map is a subset of the global map which is extractedaccording to information related to the position of the informationprocessing device in the real space. Therefore, when sharing the globalmap among a plurality of users, an increase in communication costs forthe update process of the global map is suppressed.

Further, according to the embodiment, by application of the SLAMtechnique, the local map is generated based on an input image from animaging device and feature data indicating features of appearances ofobjects. Therefore, even when the information processing device isprovided only with a monocular camera, the global map can be updated bydetecting the positions of objects with high accuracy.

Further, according to the embodiment, the global map and the local mapinclude position data of objects the real space and time stamp relatedto the position data. This allows prevention of contention for updateamong a plurality of users by comparison of the time stamp and enablesviewing of the past position data designated by a user, for example.

The series of processes by the information processing device 100described in this specification is typically implemented using software.A program composing the software that implements the series of processesmay be prestored in a storage medium mounted internally or externally toeach device, for example. Then, each program is read into a randomaccess memory (RAM) and executed by a processor such as a centralprocessing unit (CPU).

Although preferred embodiments of the present invention are described indetail above with reference to the appended drawings, the presentinvention is not limited thereto. It should be understood by thoseskilled in the art that various modifications, combinations,sub-combinations and alterations may occur depending on designrequirements and other factors insofar as they are within the scope ofthe appended claims or the equivalents thereof.

1-14. (canceled)
 15. An information processing system comprising: aserver device including: at least one circuit configured to: store aglobal map representing positions of one or more feature points onobjects in a real space, wherein the global map is generated based ondata from a plurality of information processing apparatuses; distributethe global map to an information processing apparatus of the pluralityof information processing apparatuses; update the global map based ondata from the information processing apparatus; and distribute theupdated global map to the information processing apparatus; and theinformation processing apparatus including: an imaging device thatcaptures the objects in the real space; and at least one circuitconfigured to: acquire at least a part of the global map representingpositions of the one or more feature points on objects in a real space;generate a data based on an image captured by the imaging device,wherein the data includes the positions of the one or more featurepoints on the objects in the real space and timestamp indicating acaptured time of the image captured by the imaging device; and transmitthe data to the server device for updating the global map, wherein thepositions of the one or more feature points in the global map and thedata are defined as a three-dimensional position.
 16. The informationprocessing system according to claim 15, wherein a plurality of usersare in activity in the real space.
 17. The information processing systemaccording to claim 16, wherein the at least one circuit included in theinformation processing apparatus is further configured to: generate alocal map representing positions of one or more of the objectsdetectable by a device of one user among the plurality of users; updatethe global map based on position data of the one or more objectsincluded in the local map; count a number of the one or more objectsincluded in the local map; calculate a relative position of the localmap to the global map based on the number of the one or more objects andposition data of the one or more objects included in the global map andposition data of the one or more objects included in the local map. 18.The information processing system according to claim 17, wherein the atleast one circuit included in the information processing apparatus isfurther configured to: partially visualize the positions of the one ormore objects in the global map onto a screen in response to aninstruction from the one user, wherein the objects may be either nearbyto the one user or in an area remote from the one user.
 19. Theinformation processing system according to claim 17, wherein the atleast one circuit included in the information processing apparatus isfurther configured to: generate the local map based on an input imageobtained by imaging the real space using an imaging device and featuredata indicating a feature of appearance of the one or more objects. 20.The information processing system according to claim 17, wherein theinformation processing apparatus is a terminal device possessed by theone user.
 21. The information processing system according to claim 20,wherein the at least one circuit included in the information processingapparatus is further configured to: acquire at least a part of theglobal map from a server device storing the global map, and update theglobal map of the server device by transmitting position data of the oneor more objects to the server device.
 22. The information processingsystem according to claim 21, wherein the at least one circuit includedin the information processing apparatus is further configured to:acquire a part of the global map corresponding to a local areacontaining a position of the terminal device in the real space.
 23. Theinformation processing system according to claim 21 wherein the at leastone circuit included in the information processing apparatus is furtherconfigured to: acquire a part of the global map representing positionsof a predetermined number of objects located in close proximity to theterminal device.
 24. The information processing system according toclaim 17, wherein the at least one circuit included in the informationprocessing apparatus is further configured to: perform coordinateconversion from the position data of the one or more objects included inthe local map to data of a coordinate system of the global map accordingto the relative position of the local map; and update the global map byusing the position data of the one or more objects included in the localmap after the coordinate conversion.
 25. The information processingsystem according to claim 24, wherein the at least one circuit includedin the information processing apparatus is further configured to:calculate the relative position of the local map based on position dataof an immobile object included in common in the global map and the localmap.
 26. The information processing system according to claim 24,wherein the at least one circuit included in the information processingapparatus is further configured to: calculate the relative position ofthe local map so that, when converting the position data of objectsincluded in the local map into data of the coordinate system of theglobal map, a difference between the data after conversion and theposition data of the one or more objects included in the global map issmaller as a whole.
 27. The information processing system according toclaim 24, wherein the global map includes position data of each of theone or more objects in the real space in the coordinate system of theglobal map and a time stamp related to the position data.
 28. Theinformation processing system according to claim 15, wherein the objectsinclude a landmark.
 29. The information processing system according toclaim 28, wherein the landmark is an immobile object.
 30. An informationprocessing apparatus comprising: an imaging device that captures objectsin real space; and at least one circuit configured to: acquire at leasta part of a global map representing positions of one or more featurepoints on the objects in the real space; generate a data based on animage captured by the imaging device, wherein the data includes thepositions of the one or more feature points on the objects in the realspace and timestamp indicating a captured time of the image captured bythe imaging device; and transmit the data to a server device forupdating the global map, wherein the positions of the one or morefeature points in the global map and the data are defined as athree-dimensional position.
 31. A server device comprising: at least onecircuit configured to: store a global map representing positions of oneor more feature points on objects in a real space, wherein the globalmap is generated based on data from a plurality of informationprocessing apparatuses; distribute the global map to an informationprocessing apparatus of the plurality of information processingapparatuses; update the global map based on data from the informationprocessing apparatus; and distribute the updated global map to theinformation processing apparatus, wherein the data from plurality ofinformation processing apparatus is generated based on an image capturedby an imaging device, and includes positions of one or more featurepoints on the objects in the real space and timestamp indicating acaptured time of the image captured by the imaging device, and whereinthe positions of the one or more feature points in the global map andthe data are defined as a three-dimensional position.
 32. An informationprocessing method comprising: storing, by a server device, a global maprepresenting positions of one or more feature points on objects in areal space, wherein the global map is generated based on data from aplurality of information processing apparatuses; distributing, by theserver device, the global map to an information processing apparatus ofthe plurality of information processing apparatuses; updating, by theserver device, the global map based on data from the informationprocessing apparatus; distributing, by the server device, the updatedglobal map to the information processing apparatus; capturing, by animaging device including in an information processing apparatus, theobjects in the real space; acquiring, by the information processingapparatus, at least a part of the global map representing positions ofthe one or more feature points on objects in a real space; generating,by the information processing apparatus, a data based on an imagecaptured by the imaging device, wherein the data includes the positionsof the one or more feature points on the objects in the real space andtimestamp indicating a captured time of the image captured by theimaging device; and transmitting, by the information processingapparatus, the data to the server device for updating the global map,wherein the positions of the one or more feature points in the globalmap and the data are defined as a three-dimensional position.
 33. Anon-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute a method, the method comprising: storing, by a server device, aglobal map representing positions of one or more feature points onobjects in a real space, wherein the global map is generated based ondata from a plurality of information processing apparatuses;distributing, by the server device, the global map to an informationprocessing apparatus of the plurality of information processingapparatuses; updating, by the server device, the global map based ondata from the information processing apparatus; distributing, by theserver device, the updated global map to the information processingapparatus; capturing, by an imaging device including in an informationprocessing apparatus, the objects in the real space; acquiring, by theinformation processing apparatus, at least a part of the global maprepresenting positions of the one or more feature points on objects in areal space; generating, by the information processing apparatus, a databased on an image captured by the imaging device, wherein the dataincludes the positions of the one or more feature points on the objectsin the real space and timestamp indicating a captured time of the imagecaptured by the imaging device; and transmitting, by the informationprocessing apparatus, the data to the server device for updating theglobal map, wherein the positions of the one or more feature points inthe global map and the data are defined as a three-dimensional position.